Rethinking the assessment of risk of bias due to selective reporting:a cross-sectional study

BACKGROUND: Selective reporting is included as a core domain of Cochrane’s tool for assessing risk of bias in randomised trials. There has been no evaluation of review authors’ use of this domain. We aimed to evaluate assessments of selective reporting in a cross-section of Cochrane reviews and to outline areas for improvement. METHODS: We obtained data on selective reporting judgements for 8434 studies included in 586 Cochrane reviews published from issue 1–8, 2015. One author classified the reasons for judgements of high risk of selective reporting bias. We randomly selected 100 reviews with at least one trial rated at high risk of outcome non-reporting bias (non-/partial reporting of an outcome on the basis of its results). One author recorded whether the authors of these reviews incorporated the selective reporting assessment when interpreting results. RESULTS: Of the 8434 studies, 1055 (13 %) were rated at high risk of bias on the selective reporting domain. The most common reason was concern about outcome non-reporting bias. Few studies were rated at high risk because of concerns about bias in selection of the reported result (e.g. reporting of only a subset of measurements, analysis methods or subsets of the data that were pre-specified). Review authors often specified in the risk of bias tables the study outcomes that were not reported (84 % of studies) but less frequently specified the outcomes that were partially reported (61 % of studies). At least one study was rated at high risk of outcome non-reporting bias in 31 % of reviews. In the random sample of these reviews, only 30 % incorporated this information when interpreting results, by acknowledging that the synthesis of an outcome was missing data that were not/partially reported. CONCLUSIONS: Our audit of user practice in Cochrane reviews suggests that the assessment of selective reporting in the current risk of bias tool does not work well. It is not always clear which outcomes were selectively reported or what the corresponding risk of bias is in the synthesis with missing outcome data. New tools that will make it easier for reviewers to convey this information are being developed. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1186/s13643-016-0289-2) contains supplementary material, which is available to authorized users

Personal financial incentives for changing habitual health-related behaviors: A systematic review and meta-analysis.

OBJECTIVES: Uncertainty remains about whether personal financial incentives could achieve sustained changes in health-related behaviors that would reduce the fast-growing global non-communicable disease burden. This review aims to estimate whether: i. financial incentives achieve sustained changes in smoking, eating, alcohol consumption and physical activity; ii. effectiveness is modified by (a) the target behavior, (b) incentive value and attainment certainty, (c) recipients' deprivation level. METHODS: Multiple sources were searched for trials offering adults financial incentives and assessing outcomes relating to pre-specified behaviors at a minimum of six months from baseline. Analyses included random-effects meta-analyses and meta-regressions grouped by timed endpoints. RESULTS: Of 24,265 unique identified articles, 34 were included in the analysis. Financial incentives increased behavior-change, with effects sustained until 18months from baseline (OR: 1.53, 95% CI 1.05-2.23) and three months post-incentive removal (OR: 2.11, 95% CI 1.21-3.67). High deprivation increased incentive effects (OR: 2.17; 95% CI 1.22-3.85), but only at >6-12months from baseline. Other assessed variables did not independently modify effects at any time-point. CONCLUSIONS: Personal financial incentives can change habitual health-related behaviors and help reduce health inequalities. However, their role in reducing disease burden is potentially limited given current evidence that effects dissipate beyond three months post-incentive removal.This research was funded by the Wellcome Trust as part of a Strategic Award in Biomedical Ethics; program title: “The Centre for the Study of Incentives in Health”; grant number: 086031/Z/08/Z; PI Prof. TM Marteau. The funder did not contribute to any part of this research.This is the final version of the article. It first appeared from Elsevier via http://dx.doi.org/10.1016/j.ypmed.2015.03.00

Between-trial heterogeneity in meta-analyses may be partially explained by reported design characteristics.

OBJECTIVE: We investigated the associations between risk of bias judgments from Cochrane reviews for sequence generation, allocation concealment and blinding, and between-trial heterogeneity. STUDY DESIGN AND SETTING: Bayesian hierarchical models were fitted to binary data from 117 meta-analyses, to estimate the ratio λ by which heterogeneity changes for trials at high/unclear risk of bias compared with trials at low risk of bias. We estimated the proportion of between-trial heterogeneity in each meta-analysis that could be explained by the bias associated with specific design characteristics. RESULTS: Univariable analyses showed that heterogeneity variances were, on average, increased among trials at high/unclear risk of bias for sequence generation (λˆ 1.14, 95% interval: 0.57-2.30) and blinding (λˆ 1.74, 95% interval: 0.85-3.47). Trials at high/unclear risk of bias for allocation concealment were on average less heterogeneous (λˆ 0.75, 95% interval: 0.35-1.61). Multivariable analyses showed that a median of 37% (95% interval: 0-71%) heterogeneity variance could be explained by trials at high/unclear risk of bias for sequence generation, allocation concealment, and/or blinding. All 95% intervals for changes in heterogeneity were wide and included the null of no difference. CONCLUSION: Our interpretation of the results is limited by imprecise estimates. There is some indication that between-trial heterogeneity could be partially explained by reported design characteristics, and hence adjustment for bias could potentially improve accuracy of meta-analysis results

Data extraction methods for systematic review (semi)automation: Update of a living systematic review [version 2; peer review: 3 approved]

Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search PubMed, ACL Anthology, arXiv, OpenAlex via EPPI-Reviewer, and the dblp computer science bibliography. Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This living review update includes publications up to December 2022 and OpenAlex content up to March 2023. Results: 76 publications are included in this review. Of these, 64 (84%) of the publications addressed extraction of data from abstracts, while 19 (25%) used full texts. A total of 71 (93%) publications developed classifiers for randomised controlled trials. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. Data are available from 25 (33%), and code from 30 (39%) publications. Six (8%) implemented publicly available tools Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of literature review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. Between review updates, trends for sharing data and code increased strongly: in the base-review, data and code were available for 13 and 19% respectively, these numbers increased to 78 and 87% within the 23 new publications. Compared with the base-review, we observed another research trend, away from straightforward data extraction and towards additionally extracting relations between entities or automatic text summarisation. With this living review we aim to review the literature continually

Implementing informative priors for heterogeneity in meta-analysis using meta-regression and pseudo data.

Many meta-analyses combine results from only a small number of studies, a situation in which the between-study variance is imprecisely estimated when standard methods are applied. Bayesian meta-analysis allows incorporation of external evidence on heterogeneity, providing the potential for more robust inference on the effect size of interest. We present a method for performing Bayesian meta-analysis using data augmentation, in which we represent an informative conjugate prior for between-study variance by pseudo data and use meta-regression for estimation. To assist in this, we derive predictive inverse-gamma distributions for the between-study variance expected in future meta-analyses. These may serve as priors for heterogeneity in new meta-analyses. In a simulation study, we compare approximate Bayesian methods using meta-regression and pseudo data against fully Bayesian approaches based on importance sampling techniques and Markov chain Monte Carlo (MCMC). We compare the frequentist properties of these Bayesian methods with those of the commonly used frequentist DerSimonian and Laird procedure. The method is implemented in standard statistical software and provides a less complex alternative to standard MCMC approaches. An importance sampling approach produces almost identical results to standard MCMC approaches, and results obtained through meta-regression and pseudo data are very similar. On average, data augmentation provides closer results to MCMC, if implemented using restricted maximum likelihood estimation rather than DerSimonian and Laird or maximum likelihood estimation. The methods are applied to real datasets, and an extension to network meta-analysis is described. The proposed method facilitates Bayesian meta-analysis in a way that is accessible to applied researchers. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd

Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study.

Flaws in the design of randomized trials may bias intervention effect estimates and increase between-trial heterogeneity. Empirical evidence suggests that these problems are greatest for subjectively assessed outcomes. For the Risk of Bias in Evidence Synthesis (ROBES) Study, we extracted risk-of-bias judgements (for sequence generation, allocation concealment, blinding, and incomplete data) from a large collection of meta-analyses published in the Cochrane Library (issue 4; April 2011). We categorized outcome measures as mortality, other objective outcome, or subjective outcome, and we estimated associations of bias judgements with intervention effect estimates using Bayesian hierarchical models. Among 2,443 randomized trials in 228 meta-analyses, intervention effect estimates were, on average, exaggerated in trials with high or unclear (versus low) risk-of-bias judgements for sequence generation (ratio of odds ratios (ROR) = 0.91, 95% credible interval (CrI): 0.86, 0.98), allocation concealment (ROR = 0.92, 95% CrI: 0.86, 0.98), and blinding (ROR = 0.87, 95% CrI: 0.80, 0.93). In contrast to previous work, we did not observe consistently different bias for subjective outcomes compared with mortality. However, we found an increase in between-trial heterogeneity associated with lack of blinding in meta-analyses with subjective outcomes. Inconsistency in criteria for risk-of-bias judgements applied by individual reviewers is a likely limitation of routinely collected bias assessments. Inadequate randomization and lack of blinding may lead to exaggeration of intervention effect estimates in randomized trials

Immunogenicity and seroefficacy of 10-valent and 13-valent pneumococcal conjugate vaccines: a systematic review and network meta-analysis of individual participant data

Background: Vaccination of infants with pneumococcal conjugate vaccines (PCV) is recommended by the World Health Organization. Evidence is mixed regarding the differences in immunogenicity and efficacy of the different pneumococcal vaccines. Methods: In this systematic-review and network meta-analysis, we searched the Cochrane Library, Embase, Global Health, Medline, clinicaltrials.gov and trialsearch.who.int up to February 17, 2023 with no language restrictions. Studies were eligible if they presented data comparing the immunogenicity of either PCV7, PCV10 or PCV13 in head-to-head randomised trials of young children under 2 years of age, and provided immunogenicity data for at least one time point after the primary vaccination series or the booster dose. Publication bias was assessed via Cochrane's Risk Of Bias due to Missing Evidence tool and comparison-adjusted funnel plots with Egger's test. Individual participant level data were requested from publication authors and/or relevant vaccine manufacturers. Outcomes included the geometric mean ratio (GMR) of serotype-specific IgG and the relative risk (RR) of seroinfection. Seroinfection was defined for each individual as a rise in antibody between the post-primary vaccination series time point and the booster dose, evidence of presumed subclinical infection. Seroefficacy was defined as the RR of seroinfection. We also estimated the relationship between the GMR of IgG one month after priming and the RR of seroinfection by the time of the booster dose. The protocol is registered with PROSPERO, ID CRD42019124580. Findings: 47 studies were eligible from 38 countries across six continents. 28 and 12 studies with data available were included in immunogenicity and seroefficacy analyses, respectively. GMRs comparing PCV13 vs PCV10 favoured PCV13 for serotypes 4, 9V, and 23F at 1 month after primary vaccination series, with 1.14- to 1.54- fold significantly higher IgG responses with PCV13. Risk of seroinfection prior to the time of booster dose was lower for PCV13 for serotype 4, 6B, 9V, 18C and 23F than for PCV10. Significant heterogeneity and inconsistency were present for most serotypes and for both outcomes. Two-fold higher antibody after primary vaccination was associated with a 54% decrease in risk of seroinfection (RR 0.46, 95% CI 0.23–0.96). Interpretation: Serotype-specific differences were found in immunogenicity and seroefficacy between PCV13 and PCV10. Higher antibody response after vaccination was associated with a lower risk of subsequent infection. These findings could be used to compare PCVs and optimise vaccination strategies. Funding: The NIHR Health Technology Assessment Programme